This has now changed significantly, such that it should warrant another review. The first commit has remained the same, but the second has changed.

• In order to have a true "runtimeless" executable, we can't emit segmented stacks. By emitting segmented stacks, we currently then place a dependence on libmorestack, which itself has a dependence on librustrt for allocating new stacks, and the whole point of this is to get away from the runtime. For this reason, I added a new -Z no-segstk option to disable LLVM emission of segmented stacks.
• At the same time, I also trimmed down the default linker arguments hardcoded into the compiler. Most of these arguments are just used by librustrt, and that should only be required by libstd, nothing else. I moved all of the linker options into cfg'd off modules with empty extern blocks for each target os in std.rs.
• The remaining default linker arguments, however, are those for segmented stacks. Because this actually is a compiler thing, I left the default linking to libmorestack. As a dependence, segmented stacks being enabled also force being linked to rustrt to get the stack allocation functions.
• As a final addendum, right now C functions can't be called unless librustrt is present. I'm not sure if this will change in the near future, but for now I've xfail'd the smallest-hello-world because it calls a C function which then places a dependence on librustrt (which I'm trying to avoid). In the meantime, the no-std-xcrate tests should exercise optional lang items and small executables without librustrt.

The __morestack calls are also required for stack safety, so I think there would need to be an alternative if they were removed. The calls could still be emitted to check for overflow without using segmented stacks.

@thestinger, I'm confused how that's relevant. If you inform the compiler that you don't want segmented stacks, then it's not an issue of "stack safety" but rather you're just responsible for allocating enough stack for the process. Things like tasks probably won't work, but if you're not using segmented stacks you're probably not using librustrt either.

Things like C/C++ don't deal with this as far as I know. I thought they relied on the OS giving the initial thread enough stack space and then pthreads/whatever takes care of allocating stacks for spawned threads.

At least in the context of something like a kernel, __morestack doesn't really make a lot of sense because that just indicates that everything should die.

A stack overflow is a security vulnerability, whether or not you're able to grow the stack is a separate issue. Linux distributions have stack smashing protection for C in both user space and kernel space. It's not an absolute guarantee of protection, but it does catch almost all overflows.

Right now Rust needs __morestack to protect against stack smashing vulnerabilities. It would be more efficient to use a combination of guard pages and checks on function calls allocating more stack space than the guard size, but as of right now it's not implemented. I don't think removing these checks is ever the right thing to do.

Ending the process, even if it means taking down the system, is the right thing to do when the alternative is a remote attacker being able to run a payload.

Yes, stack overflow is a vulnerability, but is this really up to the compiler to set up these checks? This is a runtime issue, not a compiler issue. If you choose to not have librustrt/__morestack, then you're suddenly taking on the responsibility of the #[start] function which can certainly set up guard pages or do whatever it likes before it actually executes the program in question.

Rust as a language does not need __morestack; the current runtime requires __morestack, however. If you opt out of using the runtime, then you shouldn't be forced to use __morestack or segmented stacks at all. It seems to me that the same logic for rust requiring __morestack would apply just as well to C++/C requiring __morestack, which they don't. Sure you're exposing a "vulnerability", but you're also already treading in far more dangerous waters by opting out of the default runtime and most likely having lots of initial very unsafe code before transitioning to safe code.

Rust as a language does need the __morestack calls inserted by the compiler to be safe from stack overflows. It's not relevant whether or not you're using segmented stacks. Guard pages alone do not provide safety, you need checks inserted by the compiler for anything larger than the guard pages.

C and C++ do use something like __morestack to be safe from stack overflows too. In the Windows world it's done with the combination of a __chkstk function and a guard page. Linux distributions rely on stack smashing protection, and many of them (including Ubuntu) have it force-enabled in the C compiler.

If we omit the __morestack calls, we'll be vulnerable to an exploit that is solved for C implementations on the platforms Rust runs on.

If you do turn off stack overflow protection, all code would have to be in unsafe functions. I think it makes more sense to turn it off on a function-by-function basis if you know the function's stack consumption is always smaller than the size of the guard pages you've set up for the thread.

The above strategy should be automatic though, similar to how Windows does it but just for the safety aspect.

Huh, I did not know that windows/linux had those enabled by default in the compilers. I looked into these and came up with:

• Windows only checks when local variables exceed 4K/8K. This is because windows will actually keep auto-growing your stack, but only if it hits the guard page at the end of the stack (exceeding 4K/8K can skip the guard page entirely).
• Linux doesn't do the same thing, the only stack protection I could find was calls to __stack_chk_fail when I compiled something on ubuntu and googled around. This looks like it's detecting buffer overflow type situations and doesn't have much do do with stack growth. Am I not finding the relevant thing here?
• OSX I couldn't find much info on, but I imagine that it's similar-ish to linux.

If I'm right about linux (which I'm probably not...), then windows is the only platform to auto-grow the stack or place checks to make sure you're not growing it too much. On linux, a program like:

#include <string.h>

int main() {
  char a[100000000];
  memset(a, 1, sizeof(a));
  return 0;
}

(when compiled without optimizations) just segfaults in calling memset because you can't push a return address. That was compiled on ubuntu and had the __stack_chk_fail function, but there's no protection for just growing the stack. If that were a rust function, in theory it would call __morestack (detecting it didn't have enough stack), and then allocate enough stack for itself.

Basically, it still seems to me like there should be a way to turn off segmented stacks/__morestack. If rust wants to be a replacement for C in the systems language space, then there should be a way to turn them off because some environments may not want them at all.

I think that there's also two things at play here. One is "I've run out of stack, can you automatically give me some more?" and the other is "Can you just tell me when I run out of stack?". Right now librustrt/__morestack implement the first, and this option of -Z no-segstk jumps over the second and goes straight to disable anything. So perhaps we want three modes of operation:

1. No extra options passed to the compiler, you're forced to use librustrt/__morestack to guarantee a program will always have enough stack.
2. -Z no-segstk disables segmented stacks, but applies the windows-style functionality. The idea here is that you don't need to worry about stack size. It's assumed that the OS/runtime have allocated one page at the end of the stack to fault on and be auto-allocated, and then all functions which require more than one page of stack will have a prologue check at the beginning to a user-defined function (which could even be called __chkstk).
3. -Z no-segstk -Z no-stackprotect disables segmented stacks and also disables the auto-inserted stack protections. This is meant for any use case which just deals with stacks on its own instead of having runtime checks to ensure there's enough stack space.

I'm not entirely sure that we want 2 to be an option, because it sounds a lot like the user just being required to define __morestack instead of providing an implementation by default.

In retrospect, this is quickly getting out of hand and should probably move to the mailing list. I think that there's a few things being discussed here and they're overlapping a bit. I'm going to think a bit more about these and maybe post something to the mailing list.

@alexcrichton: What I mean by copying the Windows functionality is marking N guard pages and then having the compiler insert the __morestack check only on functions allocating a larger amount than N * page_size. Hitting the guard page could be handled with signals and sigaltstack on *nix, and call into __morestack.

If you didn't want segmented stacks, you could replace __morestack with abort, but you would still always be 100% safe from an overflow vulnerability.

That's what I was intending by number 2 above. There's still the limitation that the stack limit has to be stored somewhere, and not only somewhere but exactly where LLVM is expecting it to be stored, so that's why I think that option 3 should still exist (to not emit this kind of code at all).

Are you ok with those three options for emission of "segmented stack" code? I think that implementation-wise no-segstk would actually mean "don't link libmorestack by default, I will provide my own implementation" and the option should probably be renamed. Option 3 would involve both not linking against libmorestack and disabling segmented stacks in LLVM.

I don't want this to rot, so I'm xfail-test flagging the test which is having linking problems so the optional lang-items fix can get checked in.

Dealing with a runtime-less rust should happen at a later time.